Investigation assist system and investigation assist method

ABSTRACT

An investigation assist system includes a plurality of servers and an integration server communicatively connected to a terminal and the plurality of servers. In response to reception of a video captured by a plurality of cameras, each of the plurality of servers performs a video analysis of an object with respect to an incident, the plurality of servers processing different objects, respectively. Based on an input of a plurality of different object feature elements from the terminal, the integration server sends a search request for corresponding objects to the respective servers corresponding to the object feature elements, receives and integrates search results of the corresponding objects from the respective servers, and causes the terminal to display an integrated search result.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present disclosure relates to an investigation assist system, aninvestigation assist method, and a computer program that assist aninvestigation such as an incident by an investigation agency.

2. Background Art

A technique is known in which a plurality of camera devices are arrangedat predetermined positions on the travel route of a vehicle, and thecamera image information captured by each camera device is displayed ona display device in a terminal device mounted on the vehicle through anetwork and a wireless information exchange device (for example, seeJP-A-2007-174016). According to JP-A-2007-174016, a user can obtain areal-time camera image having a large amount of information, based onthe camera image information captured by the plurality of camerasarranged on the traveling route of the vehicle.

SUMMARY OF THE INVENTION

In JP-A-2007-174016, since the camera image information captured by eachof the plurality of camera devices can be displayed on the displaydevice in the terminal device mounted on the vehicle, the user (forexample, a driver) can confirm real-time camera image information at thelocation where each camera device is arranged. However, inJP-A-2007-174016, in view of the recent fact that forms of incidents oraccidents (hereinafter, referred to as “incidents”) are diversifying, itis not considered to efficiently narrow down a suspect who has caused anincident or the getaway vehicle used by the suspect for escape. A personwho witnesses an incident (that is, an eyewitness) rarely remembers thedetails of the appearance of the suspected person or the getaway vehiclein detail, and often remembers some of the characteristics of the part.However, if even one or more such partial features can be collected andsearched, there is a possibility that the efficiency of early narrowingdown of the suspect or the getaway vehicle can be improved. Inparticular, in an investigation by an investigation agency such as thepolice (especially, the initial investigation), it is often required tospecify the suspect or the getaway vehicle as soon as possible. However,even if the technique of JP-A-2007-174016 is used, if an investigator(for example, a police officer) manually checks and confirms the imagesof individual camera devices, it takes time to specify the suspect orthe getaway vehicle, which is inefficient, and therefore, there is aproblem that it is difficult to detect a suspect or a getaway vehicle atan early stage.

The present disclosure has been devised in view of the above-mentionedconventional circumstances and aims to provide an investigation assistsystem, an investigation assist method, and a computer program thatimprove the convenience of an investigation by an investigation agencysuch as the police by promptly and efficiently assisting thespecification of a suspect who has caused an incident or a getawayvehicle used by the suspect for escape.

The present disclosure provides an investigation assist system thatincludes a plurality of servers and an integration servercommunicatively connected to a terminal and the plurality of servers, inwhich in response to reception of a video captured by a plurality ofcameras, each of the plurality of servers performs a video analysis ofan object with respect to an incident, the plurality of serversprocessing different objects, respectively, and based on an input of aplurality of different object feature elements from the terminal, theintegration server sends a search request for corresponding objects tothe respective servers corresponding to the object feature elements,receives and integrates search results of the corresponding objects fromthe respective servers, and causes the terminal to display an integratedsearch result.

Further, the present disclosure provides an investigation assist methodperformed by an investigation assist system including a plurality ofservers and an integration server communicatively connected to aterminal and the plurality of servers, the investigating assist methodincluding receiving a video captured by a plurality of cameras, causinga plurality of servers to perform a video analysis of an object withrespect to an incident, the plurality of servers processing differentobjects, respectively, based on an input of a plurality of differentobject feature elements from the terminal, sending a search request forcorresponding objects to the respective servers corresponding to theobject feature elements, and receiving and integrating search results ofthe corresponding objects from the respective servers, and causing theterminal to display an integrated search result.

Further, the present disclosure provides a computer program for causingan integration server which is a computer device to realizecommunicating with a terminal and communicate with a plurality ofservers that perform an video analysis of different objects with respectto an incident or like by using videos captured by a plurality ofcameras, based on an input of a plurality of different object featureelements from the terminal, sending a search request for a correspondingobject to the server corresponding to the object feature element, andreceiving and integrating search results of the corresponding objectsfrom the respective servers, and displaying the search results on theterminal.

According to the present disclosure, it is possible to quickly andefficiently assist the specification of a suspect who has caused anincident or the like, and the getaway vehicle used by the suspect forescape, and improve the convenience of an investigation by aninvestigation agency such as the police.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a system configuration example of aninvestigation assist system according to Embodiment 1.

FIG. 2 is a block diagram showing a hardware configuration example ofvarious servers that form the investigation assist system.

FIG. 3 is a block diagram showing a hardware configuration example ofvarious terminals that constitute the investigation assist system.

FIG. 4 is a diagram showing an example of a search screen displayed on aclient terminal.

FIG. 5 is a diagram showing an example of a search screen displayed onthe client terminal.

FIG. 6 is a diagram showing an example of a person search result screendisplayed on the client terminal.

FIG. 7 is a diagram showing an example of a person search result screendisplayed on the client terminal.

FIG. 8 is a diagram showing an example of a vehicle search result screendisplayed on the client terminal.

FIG. 9 is a diagram showing an example of a vehicle search result screendisplayed on the client terminal.

FIG. 10 is a diagram showing an example of a search result screen of anOR search displayed on the client terminal.

FIG. 11 is a diagram showing an example of a search result screen of anAND search displayed on the client terminal.

FIG. 12 is a sequence diagram showing an operation procedure example intime series regarding a first investigation scenario in theinvestigation assist system according to Embodiment 1.

FIG. 13 is a sequence diagram showing an operation procedure example intime series regarding a second investigation scenario in theinvestigation assist system according to Embodiment 1.

FIG. 14 is a diagram showing an example of an alarm monitoring screendisplayed on the client terminal.

FIG. 15 is a sequence diagram showing an operation procedure example intime series regarding an image search using a live video in aninvestigation assist system according to Embodiment 2.

FIG. 16 is a sequence diagram showing an operation procedure example intime series regarding an image search using a past recorded video in theinvestigation assist system according to Embodiment 2.

FIG. 17 is a sequence diagram showing an operation procedure example intime series regarding an image search using the past recorded video inthe investigation assist system according to Embodiment 2.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENT

Hereinafter, an embodiment specifically disclosing the configuration andoperation of an investigation assist system, an investigation assistmethod, and a computer program according to the present disclosure willbe described in detail with reference to the accompanying drawings.However, more detailed description than necessary may be omitted. Forexample, detailed description of well-known matters or duplicatedescription of substantially the same configuration may be omitted. Thisis to prevent the following description from being unnecessarilyredundant and to facilitate understanding by those skilled in the art.The accompanying drawings and the following description are provided forthose skilled in the art to fully understand the present disclosure, andare not intended to limit the claimed subject matter thereby.

Hereafter, by using the videos captured by cameras installed in variousplaces in a city, an example will be described in which an investigationassist system assists the investigation of a police officer who narrowsdown and tracks a suspect who has caused an incident in the city orgetaway vehicles used by the suspect for escape.

Embodiment 1

FIG. 1 is a block diagram showing a system configuration example of aninvestigation assist system 1 according to Embodiment 1. Theinvestigation assist system 1 includes at least an artificialintelligent (AI) integration server 10, a video management server 40, aface authentication server 50, a person search server 60, an behaviordetection server 70, a vehicle search server 80, and a license platerecognition (LPR) server 90. The investigation assist system 1 mayfurther include a client terminal VW1 and a mobile terminal VW2 asviewer devices. Each of the video management server 40, the faceauthentication server 50, the person search server 60, the behaviordetection server 70, the vehicle search server 80, and the LPR server 90is connected to each of a plurality (for example, 20) of cameras C1 toC20 in a communicative manner via a network NW1. Each of theAI-integration server 10, the video management server 40, the faceauthentication server 50, the person search server 60, the behaviordetection server 70, the vehicle search server 80, and the LPR server 90may be a server of the investigation assist system 1 and may be anon-premise server in a police station or a cloud server connected to anetwork such as the Internet.

Although only one client terminal VW1 and one mobile terminal VW2 areshown in FIG. 1, a plurality of client terminals VW1 and mobileterminals VW2 may be provided. Further, the investigation assist system1 is not limited to being used only within a single police station, andmay be applied to an example in which a joint investigation is carriedout across a plurality of police stations.

Each of the cameras C1 to C20 is installed at various places in the cityfor monitoring purposes, generates captured video data of a capturedarea (in other words, a subject), and sends the video data to therespective servers (specifically, the face authentication server 50, theperson search server 60, the behavior detection server 70, the vehiclesearch server 80, and the LPR server 90) via the network NW1. In thefollowing description, the captured video data includes not only thecaptured video data itself but also the identification information ofthe camera that has captured the captured video and the information ofthe captured date and time. Further, the identification information ofthe camera may include the identification information of the cameraitself and the installation location information of the camera. Each ofthe cameras C1 to C20 may be fixedly installed on the road side of amain trunk road such as a national road or a prefectural road, or may befixedly installed near an intersection. Each of the cameras C1 to C20 isconnected to each server (specifically, the face authentication server50, the person search server 60, the behavior detection server 70, thevehicle search server 80, and the LPR server 90) in a communicativemanner via the network NW1 such as an intranet communication line. Thenetwork NW1 is configured by a wired communication line (for example, anoptical communication network using an optical fiber), but may beconfigured by a wireless communication network. The cameras C1 to C20may all be manufactured by the same manufacturer, or some of the camerasmay be manufactured by other companies. Further, in the configurationexample of FIG. 1, the captured video data of each of the cameras C1 toC20 is commonly received by the respective servers (specifically, theface authentication server 50, the person search server 60, the behaviordetection server 70, the vehicle search server 80, and the LPR server90), but the common captured video data received by each server may beall the captured video data of the cameras C1 to C20 or only thecaptured video data of some cameras.

The video management server 40 as a server is installed in, for example,a police station, and includes at least a processor 41 and a database42. Hereinafter, the database may be abbreviated as “DB”. The videomanagement server 40 stores data of processing results of the faceauthentication server 50, the person search server 60, the behaviordetection server 70, the vehicle search server 80, and the LPR server90, and stores the captured video data of each of the cameras C1 to C20.Although not shown in FIG. 1, the video management server 40 may receiveand store the captured video data of each of the cameras C1 to C20 viathe network NW1, or may receive and store the captured video data ofeach of the cameras C1 to C20 from any one of the face authenticationserver 50, the person search server 60, the behavior detection server70, the vehicle search server 80, and the LPR server 90. In addition, inresponse to a request sent from the client terminal VW1 according to theoperation of an operator in the police station or the mobile terminalVW2 according to the operation of a police officer in the field, thevideo management server 40 may read the captured video data satisfyingthe request from the database 42 and send the captured video to theclient terminal VW1 or the mobile terminal VW2.

The face authentication server 50 as a server is installed in, forexample, a police station, and includes at least a processor 51 and adatabase 52. Every time the processor 51 of the face authenticationserver 50 receives the captured video data of each of the cameras C1 toC20, the processor 51 performs a video analysis such as detecting theface of a person shown in the captured video data, and stores the videoanalysis result in the database 52. When the processor 51 of the faceauthentication server 50 detects a face image registered in blacklistdata (see later) during the video analysis, the processor 51 maygenerate an alarm notification for notifying the detection of a personwhose face image is registered in the blacklist data as a video analysisresult. The face image for which the alarm notification is to begenerated is registered in advance in the face authentication server 50,and this registration may be performed by an instruction of registrationfrom the client terminal VW1 or the mobile terminal VW2 by an operationof the operator or the like. This alarm notification is sent from theface authentication server 50 to the AI-integration server 10 each timethe alarm notification is generated. The video analysis result includes,for example, a face image of a person shown in the captured video data,the captured date and time of the captured video data used for the videoanalysis, and the identification information of the camera. Further,when the processor 51 of the face authentication server 50 receives aprocessing instruction (for example, an instruction to verify a faceimage) from the AI-integration server 10, the processor 51 verifieswhether or not the face image to be verified included in the processinginstruction is registered in the blacklist data (see later) of thedatabase 52, and stores a verification result in the database 52. Here,the blacklist data (an example of a face database) is data in whichpersonal information including a face image of a person with a criminalrecord, who has caused a past incident is registered for each incident,and is registered in the database 52. The blacklist data may beregistered in the database 52 or may be registered in another externaldatabase (not shown).

The person search server 60 as a server is installed in, for example, apolice station, and includes at least a processor 61 and a database 62.Every time the processor 61 of the person search server 60 receives thecaptured video data of each of the cameras C1 to C20, the processor 61performs a video analysis for extracting information about a person (forexample, a suspect) shown in the captured video data, and stores thevideo analysis result in the database 62. When the processor 61 of theperson search server 60 detects a person who satisfies the personattribute information (for example, information indicating the externalcharacteristics of a suspicious person) during the video analysis, theprocessor 61 may generate an alarm notification for notifying thedetection of a person who satisfies the person attribute information asa video analysis result. The person attribute information for which thealarm notification is to be generated is registered in advance in theperson search server 60, and this registration may be performed by aninstruction of registration from the client terminal VW1 or the mobileterminal VW2 by an operation of the operator or the like. This alarmnotification is sent from the person search server 60 to theAI-integration server 10 each time the alarm notification is generated.The video analysis result includes, for example, the person informationshown in the captured video data (for example, a face, gender, age,hairstyle, height, body, personal belongings, accessories of the personshown in the captured video), the captured date and time of the capturedvideo data used for the video analysis, and the identificationinformation of the camera. The processor 61 of the person search server60 stores this person information in association with the captured videodata in the database 62. This video analysis result is referred to atthe time of searching for the presence or absence of relevant personinformation, which is performed based on a processing instruction (forexample, instruction to search for personal information) sent from theAI-integration server 10 when, for example, an incident occurs.

The behavior detection server 70 as a server is installed in, forexample, a police station, and includes at least a processor 71 and adatabase 72. Every time the processor 71 of the behavior detectionserver 70 receives the captured video data of each of the cameras C1 toC20, the processor 71 performs a video analysis for detecting thepresence or absence of a predetermined action (see later) caused by atleast one person shown in the captured video data, and stores the videoanalysis result in the database 72. The video analysis result includes,for example, the content (type) of a predetermined action, the captureddate and time of the captured video data used for the video analysis,and the identification information of the camera. Here, thepredetermined action is, for example, at least one of actions that maytrigger an incident such as staggering, fight, possession of pistols,shoplifting, and the like, but is not limited to these actions. When thepredetermined action is detected, the processor 71 of the behaviordetection server 70 generates an alarm notification (see Embodiment 2)including the captured date and time and the identification informationof the camera corresponding to the captured video data in which thepredetermined action is detected, and sends the alarm notification tothe AI-integration server 10.

The vehicle search server 80 as a server is installed in, for example, apolice station, and includes at least a processor 81 and a database 82.Every time the processor 81 of the vehicle search server 80 receives thecaptured video data of each of the cameras C1 to C20, the processor 81performs a video analysis for extracting information about a vehicle(for example, a getaway vehicle) shown in the captured video data, andstores the video analysis result in the database 82. When the processor81 of the vehicle search server 80 detects a vehicle satisfying thevehicle attribute information (for example, information indicating theexternal characteristics such as a vehicle type or a vehicle color ofthe getaway vehicle) during the video analysis, the processor 81 maygenerate an alarm notification for notifying the detection of a vehiclesatisfying the vehicle attribute information as a video analysis result.The vehicle attribute information for which the alarm notification is tobe generated is registered in advance in the vehicle search server 80,and this registration may be performed by an instruction of registrationfrom the client terminal VW1 or the mobile terminal VW2 by an operationof the operator or the like. This alarm notification is sent from thevehicle search server 80 to the AI-integration server 10 each time thealarm notification is generated. The video analysis result includes, forexample, vehicle information (for example, vehicle model, vehicle type,vehicle color, license plate information in the captured video) shown inthe captured video data, the captured date and time of the capturedvideo data used for the video analysis, and the identificationinformation of the camera. The processor 81 of the vehicle search server80 stores this vehicle information in association with the capturedvideo data in the database 82. This video analysis result is referred toat the time of searching for the presence or absence of relevant vehicleinformation, which is performed based on a processing instruction (forexample, instruction to search for vehicle information) sent from theAI-integration server 10 when, for example, an incident occurs.

The LPR server 90 as a server or a license authentication server isinstalled in, for example, a police station, and includes at least aprocessor 91 and a database 92. Every time the processor 91 of the LPRserver 90 receives the captured video data of each of the cameras C1 toC20, the processor 91 performs a video analysis for extracting thelicense plate of the vehicle shown in the captured video data, andstores the video analysis result in the database 92. Upon detecting alicense plate satisfying suspicious license plate data (for example,license plate information of a vehicle on which a suspicious person hasridden) during the video analysis, the processor 91 of the LPR server 90may generate an alarm notification for notifying the detection of alicense plate satisfying suspicious license plate data as a videoanalysis result. The suspicious license plate data for which the alarmnotification is to be generated is registered in advance in the LPRserver 90, and this registration may be performed by an instruction ofregistration from the client terminal VW1 or the mobile terminal VW2 byan operation of the operator or the like. This alarm notification issent from the LPR server 90 to the AI-integration server 10 each timethe alarm notification is generated. The processor 91 of the LPR server90 verifies whether the license plate information to be verifiedincluded in the processing instruction (for example, instruction toverify the license plate) is registered in the license plate list data(see later) of the database 92 based on the processing instruction sentfrom the AI-integration server 10, and stores a verification result inthe database 92. Here, the license plate list data is data in which thelicense plate information and the information (for example, face imageand personal information) about the corresponding purchaser (in otherwords, the owner) of the vehicle are registered in advance inassociation with each other, and are registered in the database 92. Thelicense plate list data may be registered in the database 92 or may beregistered in another external database (not shown).

The client terminal VW1 is installed in, for example, a police station,is used by an operator (police officer) in the police station, and isconfigured by using, for example, a laptop or desktop personal computer(PC). For example, when an incident or the like occurs, the operatorlistens to various information (eyewitness information) with respect toan incident or the like by a telephone call from a person (eyewitness)who has notified the police station of the occurrence of the incident orthe like, and operates the client terminal VW1 to input and record thedata. The client terminal VW1 sends, for example, a processing requestfor searching for a person or a vehicle that matches or is similar tothe eyewitness information to the AI-integration server 10, receives thesearch result acquired by the AI-integration server 10 through thesearch by each server (for example, the face authentication server 50,the person search server 60, the vehicle search server 80, and the LPRserver 90) from the AI-integration server 10, and displays the searchresult (see later). In addition, when the client terminal VW1 isconnected to the video management server 40 via a network in a policestation such as a wireless LAN, the client terminal VW1 may access thevideo management server 40 to acquire desired captured video data, andreproduce and display the video data.

The mobile terminal VW2 is installed in, for example, a police station,is used by a police officer who is out in the field, and is configuredby using a computer such as a smartphone or a tablet terminal. Themobile terminal VW2 sends, for example, a processing request forsearching for a person or a vehicle matching or similar to theeyewitness information heard near the site to the AI-integration server10, receives the search result acquired by the AI-integration server 10through the search by each server (for example, face authenticationserver 50, person search server 60, vehicle search server 80, and LPRserver 90) from the AI-integration server 10, and displays the searchresult (see later). Further, when the mobile terminal VW2 is connectedto the video management server 40 via a network (not shown) such as awireless LAN or a mobile phone network, the mobile terminal VW2 mayaccess the video management server 40 to acquire desired captured videodata, and reproduce and display the video data.

When the AI-integration server 10 as an integration server is installedin, for example, a police station, and the processing request forsearching for a person or a vehicle is received from the client terminalVW1 or the mobile terminal VW2, a server required for searching for theprocessing request is specified. The AI-integration server 10 generatesand sends a processing instruction corresponding to the specified server(for example, the face authentication server 50, the person searchserver 60, the vehicle search server 80, the LPR server 90). Here, inthe investigation assist system 1 according to Embodiment 1, themanufacturers (makers) of the respective servers (specifically, the faceauthentication server 50, the person search server 60, the behaviordetection server 70, the vehicle search server 80, and the LPR server90) may be the same or different.

For example, when the makers of the respective servers (specifically,the face authentication server 50, the person search server 60, thevehicle search server 80, and the LPR server 90) are the same, it isconceivable that an application screen (for example, input screen ofsearch condition or verification condition) for requesting a search fromthe client terminal VW1 or the mobile terminal VW2 to each server isgenerated in a common layout unique to the manufacturer. Therefore, theoperator or the like can perform a cross-sectional search (AND search)in which a plurality of objects (for example, a person, a vehicle, aface, and a license plate) are mixed on the input screen of a singlesearch condition.

However, when the makers of the respective servers (specifically, theface authentication server 50, the person search server 60, the vehiclesearch server 80, and the LPR server 90) are not the same, anapplication screen (for example, input screen of search condition) forrequesting a search from the client terminal VW1 or the mobile terminalVW2 to a server manufactured by a different maker is generated with adifferent search algorithm or layout for each maker. In other words,when viewed from the client terminal VW1 or the mobile terminal VW2, theinput screen (application) of the verification condition to the faceauthentication server 50, the input screen (application) of the searchcondition to the person search server 60, and the input screen(application) of the search condition to the vehicle search server 80are different from each other, and for example, it is not possible toperform a cross-sectional search in which a plurality of objects (forexample, a person, a vehicle, a face, and a license plate) are mixed atone time, which reduces the convenience of the system.

Therefore, in Embodiment 1, even if the makers of the respective servers(specifically, the face authentication server 50, the person searchserver 60, the vehicle search server 80, the LPR server 90) aredifferent, upon receiving the search processing request from the clientterminal VW1 or the mobile terminal VW2, the AI-integration server 10uses a common interface (IF) for communication (access) to each serverthat is the destination of the processing request. The interfacementioned here is, for example, a common search algorithm in which thesearch algorithm for the objects used in each server is generalized, andthe AI-integration server 10 saves this common search algorithm inadvance. The AI-integration server 10 uses a common search algorithm foreach server and sends an instruction of search or verificationprocessing instruction to the corresponding server. Further, theinterface may be, for example, an interface having a common agreement orprotocol regarding communication with respective servers (specifically,the face authentication server 50, the person search server 60, thevehicle search server 80, and the LPR server 90), or may be anindividual interface suitable for communication with each server. TheAI-integration server 10 may send and receive data or information (forexample, receive an alarm notification) by using an interface suitablefor communication with each server.

The AI-integration server 10 is configured with, for example, ahigh-performance server computer, and specifically includes a memoryMM1, a processor 11, a database 12, a server IF controller 13, and aclient IF controller 14.

The memory MM1 is configured by using, for example, a random accessmemory (RAM) and a read only memory (ROM), and temporarily stores aprogram necessary to execute the operation of the AI-integration server10, and further, data or information generated during the operation. TheRAM is, for example, a work memory used when the processor 11 operates.The ROM stores in advance a program for controlling the processor 11,for example. The memory MM1 records road map information indicating thepositions where the cameras C1 to C20 are installed, and records theinformation of the updated road map every time the information of theroad map is updated due to, for example, new construction or maintenancework of the road.

The processor 11 is configured by using, for example, a centralprocessing unit (CPU), a digital signal processor (DSP) or a fieldprogrammable gate array (FPGA), functions as a control unit of theAI-integration server 10, and performs control processing of generallycontrolling the operation of each part of the AI-integration server 10,data input/output processing with respect to each part of theAI-integration server 10, data calculation processing, and data storageprocessing. The processor 11 operates in accordance with a computerprogram according to the present disclosure stored in the memory MM1.This computer program causes, for example, the AI-integration server 10which is a computer device to realize a step of communicating with aterminal (for example, the client terminal VW1 or the mobile terminalVW2), a step of communicating with a plurality of servers (for example,the face authentication server 50, the person search server 60, thebehavior detection server 70, the vehicle search server 80, and the LPRserver 90) that perform a video analysis of different objects withrespect to an incident or the like by using the captured video data ofeach of the plurality of cameras C1 to C20, a step of sending a searchrequest for a corresponding object to a server corresponding to anobject feature element based on an input of a plurality of differentobject feature elements from the terminal, and a step of receivingsearch results of the corresponding objects from the respective serversto integrate and display the search results on the terminal.

For example, upon receiving a processing request for searching for aperson or a vehicle from the client terminal VW1 or the mobile terminalVW2, the processor 11 specifies at least one server required forsearching for the processing request. The processor 11 generates andsends a processing instruction (for example, a search instruction)corresponding to the specified server (for example, the faceauthentication server 50, the person search server 60, the vehiclesearch server 80, and the LPR server 90). Thereby, the processor 11 canperform a cross-sectional search for a person and a vehicle by usingeach of a plurality of servers (for example, the person search server 60and the vehicle search server 80) (cross-sectional search processing).For example, the processor 11 performs, as the cross-sectional searchprocessing, a narrowed-down search using the two objects of the featureand the face of the person, or the three objects of the vehicle, theperson, and the face on the corresponding server.

For example, when the processor 11 receives the search result from eachserver, the processor 11 sorts the search results for each object (forexample, a person or a vehicle) (search result sorting processing). Forexample, the processor 11 determines a rank indicating the matchingdegree of the images (for example, person thumbnails, face thumbnails,and vehicle thumbnails) included in the search result based on the score(for example, a probability value indicating the likelihood of thesearch result obtained based on the processing of an AI engine) includedin the search result from each server, and rearranges the imagesaccording to the rank.

For example, the processor 11 sends a predetermined command to each ofthe servers (specifically, the video management server 40, the faceauthentication server 50, the person search server 60, the behaviordetection server 70, the vehicle search server 80, and the LPR server90) which are connected to the AI-integration server 10. The processor11 monitors whether the server is up or down (that is, whether or not aprocess in the server computer is operating) depending on whether or nota command response is received from each server (up and down monitoringprocessing).

The database 12 is configured by using, for example, a hard disk drive(HDD) or a solid state drive (SSD), and stores data or informationacquired or generated by the processor 11.

The server IF controller 13 is configured with a communication interfacecircuit that controls communication (transmission/reception) between theAI-integration server 10 and the respective servers (specifically, theface authentication server 50, the person search server 60, the behaviordetection server 70, the vehicle search server 80, and the LPR server90). Further, the server IF controller 13 selects and uses an interfacesuitable for communication (access) to each server even if the makers ofthe face authentication server 50, the person search server 60, and thevehicle search server 80 are different. For example, the server IFcontroller 13 has an input/output interface for the face authenticationserver 50, an input/output interface for the person search server 60,and an input/output interface for the vehicle search server 80,respectively, and selects and uses an interface suitable for the searchprocessing request from the client terminal VW1 or the mobile terminalVW2.

The client IF controller 14 is configured with a communication interfacecircuit that controls communication (transmission/reception) with eachof the client terminal VW1, the mobile terminal VW2, and the videomanagement server 40. The client IF controller 14 sends the searchresults sorted by the processor 11 to the client terminal VW1 or themobile terminal VW2. The client IF controller 14 instructs the videomanagement server 40 to distribute the captured video data to the clientterminal VW1 or the mobile terminal VW2 or record the captured videodata of each of the cameras C1 to C20. Further, the client IF controller14 transfers the alarm notification from the behavior detection server70 (see Embodiment 2) to each of the terminals (specifically, the clientterminal VW1 and the mobile terminal VW2). The client IF controller 14may transfer the alarm notification from the servers (for example, theface authentication server 50, the person search server 60, the vehiclesearch server 80, and the LPR server 90) other than the behaviordetection server 70 to each of the terminals (specifically, the clientterminal VW1 and the mobile terminal VW2).

FIG. 2 is a block diagram showing an example of a hardware configurationof various servers that constitute the investigation assist system 1.Specifically, the various servers are the video management server 40,the face authentication server 50, the person search server 60, thebehavior detection server 70, the vehicle search server 80, and the LPRserver 90. Therefore, the server in the description of FIG. 2 is used asa term for collectively referring to the video management server 40, theface authentication server 50, the person search server 60, the behaviordetection server 70, the vehicle search server 80, and the LPR server90. In FIG. 2, the hardware configuration common to the respectiveservers will be described in detail, and the characteristic operation ofeach unit will be described in detail with reference to FIG. 1 or later,thus the description will be simplified here.

The server of FIG. 2 is configured with, for example, a server computer,and specifically includes a processor PRC1, a memory MM2, a database 52,a camera IF controller 53, and a server IF controller 54.

The processor PRC1 is configured by using, for example, a graphicalprocessing unit (GPU) or an FPGA, functions as a control unit of theserver, and performs control processing of generally controlling theoperation of each part of the server, data input/output processing withrespect to each part of the server, data calculation processing, anddata storage processing. The processor PRC1 operates in accordance witha program stored in the memory MM2. The processor PRC1 of respectiveservers (specifically, the face authentication server 50, the personsearch server 60, the behavior detection server 70, the vehicle searchserver 80, and the LPR server 90) can execute a learned model generatedby machine learning so as to be suitable for processing by thecorresponding server, for example. Each server outputs a processingresult and a score (see later) indicating the likelihood (confidenceprobability) of the processing result by executing the processing usingthe learned model.

For example, the face authentication server 50 uses the learned modelfor the face authentication server 50 to detect the face of a personshown in the captured video data of each of the cameras C1 to C20, andto execute the verification processing between the face image includedin the verification instruction from the AI-integration server 10 andthe blacklist data in the database 52. The face authentication server 50outputs, as a processing result, the face image registered in theblacklist data and a score indicating the likelihood of the face image.

For example, the person search server 60 uses the learned model for theperson search server 60 to detect and extract person information relatedto an object (person) shown in the captured video data of each of thecameras C1 to C20, and to execute the search processing of a person whosatisfies the person search condition included in the verificationinstruction from the AI-integration server 10 by referring to thedatabase 62. The person search server 60 outputs, as a processingresult, the thumbnail (image) of a person who satisfies the personsearch condition, person information, and a score indicating thelikelihood of the thumbnail.

For example, the behavior detection server 70 uses the learned model forthe behavior detection server 70 to detect the presence or absence of apredetermined action caused by an object (person) shown in the capturedvideo data of each of the cameras C1 to C20. The behavior detectionserver 70 outputs, as a processing result, the content (result) of thepredetermined action determined to have the highest likelihood, thecaptured date and time of the captured video data in which the action isdetected, and the identification information of the camera.

For example, the vehicle search server 80 uses the learned model for thevehicle search server 80 to detect and extract vehicle informationrelated to an object (vehicle) shown in the captured video data of eachof the cameras C1 to C20, and to execute search processing of a vehiclesatisfying the vehicle search condition included in the verificationinstruction from the AI-integration server 10 by referring to thedatabase 82. The vehicle search server 80 outputs, as a processingresult, the thumbnail (image) of a vehicle satisfying the vehicle searchcondition, vehicle information, and a score indicating the likelihood ofthe thumbnail.

For example, the LPR server 90 uses the learned model for the LPR server90 to detect and extract license plate information related to an object(license plate) shown in the captured video data of each of the camerasC1 to C20, and to execute the verification processing between thelicense plate information included in the verification instruction fromthe AI-integration server 10 and the license plate list data in thedatabase 92. The LPR server 90 outputs, as a processing result, the faceimage and personal information of the purchaser (owner) of the vehiclecorresponding to the license plate registered in the license plate listdata.

The memory MM2 is configured by using, for example, a RAM and a ROM, andtemporarily stores a program necessary to execute the operation of theserver, and further data or information generated during the operation.The RAM is a work memory used when the processor PRC1 operates, forexample. The ROM stores in advance a program for controlling theprocessor PRC1, for example.

The database 52 is configured by using, for example, an HDD or SSD, andstores data or information acquired or generated by the processor PRC1of the server. The data generated by the processor PRC1 is, for example,the person information (see above) obtained as a result of the faceimage verification processing when the server is the face authenticationserver 50 or the result of the search processing when the server is theperson search server 60, is the vehicle information (see above) obtainedas a result of the search processing when the server is the vehiclesearch server 80, and is the license plate information obtained as aresult of the search processing when the server is the LPR server 90.

The camera IF controller 53 is configured with a communication interfacecircuit that controls communication (transmission/reception) between theserver and each of the cameras C1 to C20. The camera IF controller 53receives the captured video data captured by each of the cameras C1 toC20 and outputs the captured video data to the processor PRC1.

The server IF controller 54 is configured with a communication interfacecircuit that controls communication (transmission/reception) between theserver and the AI-integration server 10. The server IF controller 54receives a processing instruction from the AI-integration server 10 andreturns the processing result of the processor PRC1 based on theprocessing instruction to the AI-integration server 10. The server IFcontroller 54 also sends an alarm notification (see above) correspondingto the object detected by the processor PRC1 of each server to theAI-integration server 10.

FIG. 3 is a block diagram showing a hardware configuration example ofvarious terminals that constitute the investigation assist system 1.Specifically, the various terminals are the client terminal VW1 and themobile terminal VW2. Therefore, the terminal in the description of FIG.3 is used as a term for collectively referring to the client terminalVW1 and the mobile terminal VW2. In FIG. 3, the hardware configurationcommon to the respective terminals will be described in detail, and thecharacteristic operation of each unit will be described in detail withreference to FIG. 1 or later, thus the description will be simplifiedhere.

The terminal of FIG. 3 is configured with, for example, a computer, andspecifically includes a processor PRC2, a memory MM3, a recording device112, a server IF controller 114, an input device 115, a display device116, and a speaker 117.

The processor PRC2 is configured by using, for example, a centralprocessing unit (CPU), a digital signal processor (DSP), or an FPGA,functions as a control unit of the terminal, and performs controlprocessing of generally controlling the operation of each part of theterminal, data input/output processing with respect to each part of theterminal, data calculation processing, and data storage processing. Theprocessor PRC2 operates in accordance with a program stored in thememory MM3.

The memory MM3 is configured by using, for example, a RAM and a ROM, andtemporarily stores a program necessary to execute the operation of theterminal, and further data or information generated during theoperation. The RAM is a work memory used when the processor PRC2operates, for example. The ROM stores in advance a program forcontrolling the processor PRC2, for example. The memory MM3 records roadmap information indicating the positions where the cameras C1 to C20 areinstalled, and records the information of the updated road map everytime the information of the road map is updated due to, for example, newconstruction or maintenance work of the road.

The recording device 112 is configured by using, for example, an HDD orSSD, and stores data or information acquired or generated by theprocessor PRC2 of the terminal. The recording device 112 stores data ofvarious search results sent from the AI-integration server 10.

The server IF controller 114 is configured with a communicationinterface circuit that controls communication (transmission/reception)between the terminal and the AI-integration server 10. The server IFcontroller 114 sends a search processing request generated by theprocessor PRC2 to the AI-integration server 10. The server IF controller114 also receives various search results (processing results) or alarmnotifications (see above) sent from the AI-integration server 10.

The input device 115 accepts an operation of an operator in the policestation (for example, the user of the client terminal VW1) or a policeofficer who is out in the field (for example, a user of the mobileterminal VW2). The input device 115 is configured with, for example, amouse, a keyboard, and a touch panel.

The display device 116 is configured with, for example, a liquid crystaldisplay (LCD) or an organic electroluminescence (EL), and displaysvarious data sent from the processor PRC2.

The speaker 117 acoustically outputs a sound when the processor PRC2reproduces data (for example, video data included in the search resultfrom the AI-integration server 10).

Next, in the police investigation using the investigation assist system1 according to Embodiment 1, examples of various screens displayed onthe display device 116 of the client terminal VW1 will be described withreference to FIGS. 4 to 11. In the description of FIGS. 4 to 11, thesame configurations as the configurations shown in the drawings arereferred to by the same reference numerals to simplify or omit thedescription. The screen examples displayed in each of FIGS. 4 to 11 maybe displayed on the mobile terminal VW2. In order to make thedescription of FIGS. 4 to 11 easy to understand, it is assumed that thescreen examples shown in FIGS. 4 to 11 are displayed on the clientterminal VW1, but the “client terminal VW1” may be read as “mobileterminal VW2”, and the “operator” may be read as “police officer”.

In the police investigation, the client terminal VW1 launches andexecutes a preliminarily installed investigation assist application(hereinafter, referred to as “investigation assist application”) by theoperation of the operator. The investigation assist application isstored in, for example, the ROM of the memory MM3 of the client terminalVW1 and is executed by the processor PRC2 when activated by theoperation of the operator. In other words, the investigation assistapplication as an operating subject in the following description can beread as the processor PRC2. The data or information generated by theprocessor PRC2 while the investigation assist application is running istemporarily stored in the RAM of the memory MM3.

FIGS. 4 and 5 are diagrams showing an example of the search screendisplayed on the client terminal VW1. The investigation assistapplication displays a search screen WD1 shown in FIG. 4 or FIG. 5 onthe display device 116 by the operation of the operator when theoperator starts a search for the person or the vehicle shown in thecaptured video data of the camera.

The search screen WD1 shown in FIG. 4 has an input field CON1 forvarious search conditions regarding an object (for example, a person)shown in the captured video data. When the person information issearched, the captured video data of one or more cameras selected ordesignated by the operation of the operator is a search target.

The input field CON1 for search conditions is provided with respectivedisplay areas so that Time & Date, Camera, Search mode, and a searchicon SC1 can be input or selected.

In the display area of the date and time (Time & Date), a date and timestart input field (From), a date and time end input field (To), and aLatest icon are arranged.

The date and time start input field (From) is input by the operator asthe start date and time when the captured video data to be searched foran object (for example, a person such as a suspect or a vehicle such asa getaway vehicle) was captured. In the date and time start input field,for example, the date and time of occurrence of an incident or the dateand time slightly before the date and time are input. In FIGS. 4 and 5,for example, “6:13 pm, Jun. 18, 2019” is input in the date and timestart input field. When input by the operation of the operator, theinvestigation assist application sets the date and time input in thedate and time start input field as a search condition (for example,start date and time).

The date and time end input field (To) is input by the operator as theend date and time when the captured video data to be searched for theobject (for example, a person such as a suspect or a vehicle such as agetaway vehicle) was captured. In the date and time end input field, forexample, a predetermined period (for example, a date and time slightlyafter the date and time of occurrence of the incident or the like) isinput from the date and time input in the date and time start inputfield. In FIGS. 4 and 5, for example, “6:23 pm, Jun. 18, 2019” is inputin the date and time end input field. When input by the operation of theoperator, the investigation assist application sets the date and timeinput in the date and time end input field as a search condition (forexample, end date and time).

The Latest icon is an icon for setting the search date and time to thelatest date and time, and when pressed by the operation of the operatorduring the investigation, the investigation assist application sets thelatest date and time (for example, a period 10 minutes before the dateand time when the button is pressed) as a search condition (for example,period).

In the display area of the camera (Camera), a selection screen (notshown) for the camera to be searched is displayed. When a camera isselected by the operation of the operator, the investigation assistapplication sets the selected camera as a search target of capturedvideo data.

In the display area of the search mode (Search mode), a selection iconof the search mode intended by the operator is arranged. For example, anEvent icon and an Analytics icon are arranged. In FIGS. 4 and 5, theAnalytics icon is selected by the operation of the operator.

The Event icon is selected, for example, when searching for data relatedto an event (case) such as a past incident.

The Analytics icon is selected, for example, when searching for anobject shown in captured video data of a camera. Upon detecting that theAnalytics icon has been selected, the investigation assist applicationdisplays a sub-window in which a People icon OB1, a Face icon OB2, aVehicle icon OB3, and an LPR icon OB4 are arranged.

The People icon OB1 is selected by the operation of the operator whensearching for a person such as a suspect as an object shown in thecaptured video data of the camera. The Face icon OB2 is selected by theoperation of the operator when requesting the verification processing ofthe face of a person such as a suspect as an object shown in thecaptured video data of the camera to the face authentication server 50.The Vehicle icon OB3 is selected by the operation of the operator whensearching for a vehicle such as a getaway vehicle as an object shown inthe captured video data of the camera. The LPR icon OB4 is selected bythe operation of the operator when requesting the verificationprocessing of the license plate of a vehicle such as a vehicle that is agetaway vehicle as an object shown in the captured video data of thecamera to the LPR server 90.

Upon detecting that the People icon OB1 has been pressed by theoperation of the operator, the investigation assist application displaysa person detail screen WD2 on the display device 116 (see FIG. 4). Theperson detail screen WD2 prompts the operator to select personcharacteristic elements (an example of an object feature element) forcharacterizing a person such as a suspect. The person characteristicelements are, specifically, Gender, Hair Style, clothes of Upper Body,clothes of Lower Body, Bag, Accessory), and the colors thereof. In FIG.4, a different color palette CLP1 is provided for each of the hairstyle,clothes of upper body, and clothes of lower body. The investigationassist application sets at least one person characteristic elementselected by the operation of the operator as a person search conditionCH1. For example, in the person search condition CH1 in FIG. 4, theobject feature elements “male”, “wearing yellow long-sleeved clothes forupper body”, “wearing black pants for lower body”, “no bag”, and “noaccessories” are selected.

The search icon SC1 is pressed by the operation of the operator when asearch using the set person search condition CH1 is started. Upondetecting that the search icon SC1 has been pressed, the investigationassist application generates a person search processing requestincluding the person search condition CH1 and sends the request to theAI-integration server 10. As a result, a request for a search (forexample, a search for a person such as a suspect) from the clientterminal VW1 to the AI-integration server 10 is started.

The search screen WD1 shown in FIG. 5 has an input field CON2 forvarious search conditions regarding an object (for example, a vehicle)shown in the captured video data. When the vehicle information issearched, the captured video data of one or more cameras selected ordesignated by the operation of the operator is a search target.

Similar to the input field CON1, the input field CON2 for searchconditions is provided with respective display areas so that Time &Date), Camera, Search mode, and the search icon SC1 can be input orselected.

Upon detecting that the Vehicle icon OB3 has been pressed by theoperation of the operator, the investigation assist application displaysa vehicle detail screen WD3 on the display device 116 (see FIG. 5). Thevehicle detail screen WD3 prompts the operator to select vehiclecharacteristic elements (examples of an object feature element) forcharacterizing a vehicle such as a getaway vehicle. The vehiclecharacteristic elements are, specifically, a vehicle type (Type) and avehicle color (Color). In FIG. 5, a plurality of options CLP2 and CLP3are provided for each of the vehicle type and the vehicle color. Theinvestigation assist application sets at least one vehiclecharacteristic element selected by the operation of the operator as avehicle search condition CH2. For example, in the vehicle searchcondition CH2 of FIG. 5, the object feature elements of “Sedan”, “Van”,“sports utility vehicle (SUV)”, and “white” are selected.

The search icon SC1 is pressed by the operation of the operator when asearch using the set vehicle search condition CH2 is started. Upondetecting that the search icon SC1 has been pressed, the investigationassist application generates a vehicle search processing requestincluding the vehicle search condition CH2 and sends the request to theAI-integration server 10. As a result, a request for searching(searching for a vehicle such as a getaway vehicle) from the clientterminal VW1 to the AI-integration server 10 is started.

FIGS. 6 and 7 are diagrams showing an example of a person search resultscreen displayed on the client terminal VW1. When the client terminalVW1 receives the result (search result) of the search processing of aperson such as a suspect from the AI-integration server 10, theinvestigation assist application displays a search result screen WD4shown in FIG. 6 or 7 on the display device 116. The search resultincludes, for example, a thumbnail for each person (that is, a personwho satisfies the person search condition CH1. The same applieshereinafter), a captured date and time of captured video data of acamera that is the source of the thumbnail, identification informationof the camera, and a score of the search processing by the person searchserver 60.

The search result screen WD4 shown in FIG. 6 includes the input fieldCON1 for various search conditions regarding an object (for example, aperson) shown in the captured video data, a display area RTPS1 ofthumbnails of one or more corresponding persons included in the searchresult, and a person detail display area PSDTL1. The person detaildisplay area PSDTL1 includes a display area of a selected person videoSMV1 and a display area of a road map data MP1.

When the client terminal VW1 receives the person search result from theAI-integration server 10, the investigation assist application displaysthumbnails THM1, THM2, THM3, THM4, THM5, THM6, THM7, THM8, THM9, andTHM10 of one or more persons included in the search result side by sidein the display area RTPS1. Here, each thumbnail displayed in the displayarea RTPS1 indicates an image cut out by the person search server 60from the captured video data of the camera so that a rough whole pictureof the person is displayed, for example. The investigation assistapplication may display the persons in the search result in descendingorder of score (for example, a probability value indicating thelikelihood of the search result obtained based on the processing of theAI engine configured by the processor 61), or may display the persons inthe order of oldest or newest captured date and time when the personswere captured.

Further, upon detecting that any one of the thumbnails THM1 to THM10(for example, the thumbnail THM1) has been selected by the operation ofthe operator (see the thick arrow in FIG. 6), the investigation assistapplication requests the video management server 40 for captured videodata satisfying the identification information of the cameracorresponding to the thumbnail THM1 and the captured date and time. Uponreceiving the captured image data (that is, the data of the selectedperson image SMV1) sent from the video management server 40 in responseto this request, the investigation assist application displays theselected person video SMV1 in the person detail display area PSDTL1.Further, the investigation assist application displays a camerainstallation location CMPS1 corresponding to the thumbnail THM1 on theroad map data MP1 in a superimposed manner in the person detail displayarea PSDTL1.

The investigation assist application may display an outer frame WK1emphasizing the whole picture of the person shown in the thumbnail THM1(that is, a male wearing yellow long-sleeved clothes and black pants) onthe selected person video SMV1 in a superimposed manner. Thereby, theoperator can determine at a glance where in the selected person videoSMV1 the person of the thumbnail THM1 is present.

The investigation assist application may display the icons indicatinginstallation locations CMPS2, CMPS3, and CMPS4 of cameras other than thecamera corresponding to the thumbnail THM1, and a snapshot CHMG1 of animage of one scene of the video data captured by the camera of theinstallation location CMPS1 on the road map data MP1 in a superimposedmanner. Thereby, the operator can easily confirm the snapshot of thescene captured by the camera showing the person of the thumbnail THM1and the installation locations of the cameras other than theinstallation location CMPS1 of the camera.

The search result screen WD4 shown in FIG. 7 includes the input fieldCON1 for various search conditions regarding an object (for example, aperson) shown in the captured video data, the display area RTPS1 ofthumbnails of one or more corresponding persons included in the searchresult, and a person detail display area PSDTL2. The person detaildisplay area PSDTL2 includes a display area of the selected person videoSMV1, a display area of the road map data MP1, and a display area of aselected person detail information MFE1. The description of the displayof the thumbnails of one or more relevant persons included in the searchresult and the description regarding the display of the selected personvideo SMV1 and the road map data MP1 have been described with referenceto FIG. 6, and therefore the description in FIG. 7 will be omitted.

Upon detecting that any one of the thumbnails THM1 to THM10 (forexample, the thumbnail THM1) has been selected by the operation of theoperator (see the thick arrow in FIG. 6), the investigation assistapplication generates a face image in which the face part of the personcorresponding to the thumbnail THM1 is cut out. The investigation assistapplication generates a verification instruction including the generatedface image data and sends the instruction to the AI-integration server10. The AI-integration server 10 sends the verification instruction sentfrom the client terminal VW1 to the face authentication server 50,receives the verification processing result of the face authenticationserver 50, and sends the result to the client terminal VW1. Theinvestigation assist application further displays the selected persondetail information MFE1 including a face image FCE1 and personalinformation MTA1 of the hit person (that is, a person who matches aperson such as a person with a criminal record registered in theblacklist data of the face authentication server 50) included in theverification processing result sent from the AI-integration server 10 inthe person detail display area PSDTL2.

FIGS. 8 and 9 are diagrams showing an example of a vehicle search resultscreen displayed on the client terminal VW1. When the client terminalVW1 receives the result (search result) of the search processing of avehicle such as a getaway vehicle from the AI-integration server 10, theinvestigation assist application displays a search result screen WD5shown in FIG. 8 or 9 on the display device 116. The search resultincludes, for example, a thumbnail for each vehicle (that is, a vehiclesatisfying the vehicle search condition CH2. The same applieshereinafter), a captured date and time of captured video data of acamera which is a source of the thumbnail, identification information ofthe camera, and a score of the search processing by the vehicle searchserver 80.

The search result screen WD5 shown in FIG. 8 includes the input fieldCON2 for various search conditions regarding an object (for example, avehicle) shown in the captured video data, a display area RTVC1 ofthumbnails of one or more corresponding vehicles included in the searchresult, an event detail display area RTVC2, and a selected vehicledetail display area RTVC3.

When the client terminal VW1 receives the vehicle search result from theAI-integration server 10, the investigation assist application displaysthumbnails of one or more corresponding vehicles (for example, 16thumbnails in FIG. 8) included in the search result side by side in thedisplay area RTVC1. Here, each thumbnail displayed in the display areaRTVC1 is, for example, an image cut out by the vehicle search server 80from the captured video data of the camera so that a rough overall imageof the vehicle is displayed. The investigation assist application maydisplay the vehicles in the search result in descending order of score(for example, a probability value indicating the likelihood of thesearch result obtained based on the processing of the AI engine), ordisplay the vehicles in order of oldest or newest captured date and timewhen the vehicles were captured. The investigation assist applicationdisplays event information including at least the captured date andtime, name, and license plate of the camera that has captured eachcorresponding vehicle (an example of the event) in the event detaildisplay area RTVC2 along with the display of the thumbnail of eachcorresponding vehicle in the display area RTVC1.

Further, upon detecting that any one (for example, the thumbnail THM11)has been selected by the operation of the operator (see the thick arrowin FIG. 8), the investigation assist application requests the videomanagement server 40 for captured video data satisfying theidentification information of the camera corresponding to the thumbnailTHM11 and the captured date and time. Upon receiving that the capturedvideo data (that is, the data of the selected vehicle video LPcap1) sentfrom the video management server 40 in response to this request, theinvestigation assist application displays the data of the selectedvehicle video LPcap1 in the selected vehicle detail display area RTVC3.Further, the investigation assist application highlights an eventinformation INF1 corresponding to the vehicle selected by the thumbnailTHM11.

The investigation assist application displays not only the data of theselected vehicle video LPcap1 but also a license plate detail displayarea PLT1 including a license plate image LPcap2 of the selected vehicleand a detail display area VCL1 of the selected vehicle in the selectedvehicle detail display area RTVC3. The license plate image LPcap2 is,for example, an image in which the license plate part shown in the dataof the selected vehicle video LPcap1 is cut out by the video managementserver 40 or the client terminal VW1. Therefore, the operator canconfirm details such as the image of the vehicle, the license plate, thevehicle type, and the vehicle color of the vehicle in question (see thethick arrow in FIG. 8) among the vehicles that satisfy the vehiclesearch condition CH2 at a glance by browsing the search result screenshown in FIG. 8.

The search result screen WD5 shown in FIG. 9 includes the input fieldCON2 for various search conditions regarding an object (for example, avehicle) shown in the captured video data, the display area RTVC1 ofthumbnails of one or more corresponding vehicles included in the searchresult, the event detail display area RTVC2, and a selected vehicledetail display area RTVC3 a. The selected vehicle detail display areaRTVC3 a further includes a selected vehicle owner detail display area inaddition to the data display area of the selected vehicle video LPcap1,the license plate detail display area PLT1, and the detail display areaVCL1 of the selected vehicle. The description of the thumbnail displayarea RTVC1, the event detail display area RTVC2, the data display areaof the selected vehicle video LPcap1, the license plate detail displayarea PLT1, and the detail display area VCL1 of the selected vehicle havebeen described with reference to FIG. 8, and therefore the descriptionin FIG. 9 will be omitted.

Upon detecting that one (for example, the thumbnail THM11) has beenselected by the operation of the operator (see the thick arrow in FIG.9), the investigation assist application extracts the license plateinformation from the vehicle information corresponding to the vehiclecorresponding to the thumbnail THM11. The investigation assistapplication generates a verification instruction including the extractedlicense plate information and sends the instruction to theAI-integration server 10. The AI-integration server 10 sends theverification instruction sent from the client terminal VW1 to the LPRserver 90, receives the verification processing result of the LPR server90, and sends the result to the client terminal VW1. The investigationassist application displays a face image FCE2 and personal informationMTA2 of the purchaser (that is, the purchaser of the vehiclecorresponding to the license plate information that matches the licenseplate information registered in the license plate list data of the LPRserver 90) of the hit vehicle included in the verification processingresult sent from the AI-integration server 10 in the selected vehicleowner detail display area.

Further, in Embodiment 1, the investigation assist application canselectively cause the AI-integration server 10 to perform a search(so-called AND search) using search conditions that encompass aplurality of different objects (that is, satisfies all), or a search(so-called OR search) that adds up the results of individually searchinga plurality of different objects. Which of the AND search and the ORsearch is to be executed can be selected, for example, by the operationof the operator.

FIG. 10 is a diagram showing an example of a search result screen of anOR search displayed on the client terminal VW1. A search result screenWD6 shown in FIG. 10 shows a search result of an OR search for logicallyadding the results of individually searching a plurality of differentobjects (specifically, a person and a vehicle). Specifically, the searchresult screen WD6 shows the search result including both the searchresult of the search result screen WD4 shown in FIG. 6 (that is, see thedisplay area RTPS1 of thumbnails of one or more corresponding persons)and the search result of the search result screen WD5 shown in FIG. 8(that is, see the display area RTVC1 of thumbnails of one or morecorresponding vehicles). The search result screen WD6 shown in FIG. 10includes at least an input field CON3 for various search conditionsregarding a plurality of different objects (for example, persons andvehicles) shown in the captured video data, the display area RTPS1 ofthumbnails of one or more corresponding persons included in the searchresult, and a person detail display area PSDTL1 a.

In the input field CON3 for search conditions, both the person searchcondition CH1 (see FIG. 4) and the vehicle search condition C112 (seeFIG. 5) are selected and set by the operation of the operator in orderto search for each of a plurality of different objects (specifically,persons and vehicles). In FIG. 10, since the content of the personsearch condition CH1 is the same as the person search condition CH1shown in FIG. 4, and the content of the vehicle search condition CH2 isthe same as the vehicle search condition CH2 shown in FIG. 5, detaileddescription thereof will be omitted.

Upon detecting that a thumbnail of a person (for example, the thumbnailTHM1) is selected from a plurality of thumbnails of a person and aplurality of thumbnails of a vehicle (see the thick arrow in FIG. 10),as in the case of FIG. 6, the investigation assist application requeststhe video management server 40 for captured video data satisfying theidentification information of the camera corresponding to the thumbnailTHM1 and the captured date and time. Upon receiving the captured videodata (that is, the data of the selected person video SMV1) sent from thevideo management server 40 in response to this request, theinvestigation assist application displays the selected person video SMV1in the person detail display area PSDTL1 a. Further, the investigationassist application displays the installation location CMPS1 of thecamera corresponding to the thumbnail THM1 and the snapshot CPIMG1 of animage of one scene of the captured video data of the camera of theinstallation location CMPS1 on the road map data MP1 in a superimposedmanner in the person detail display area PSDTL1 a. Thereby, the operatorcan easily confirm the snapshot of the scene captured by the camerashowing the person of the thumbnail THM1 and the installation locationof the camera CMPS1.

On the other hand, although not shown in FIG. 10, upon detecting that avehicle thumbnail (for example, the thumbnail THM11) has been selectedfrom the plurality of thumbnails of a person and the plurality ofthumbnails of a vehicle, as in the case of FIG. 8, the investigationassist application requests the video management server 40 for capturedvideo data satisfying the identification information of the cameracorresponding to the thumbnail THM11 and the captured date and time.Upon receiving the captured video data (that is, the data of theselected vehicle video LPcap1) sent from the video management server 40in response to this request, the investigation assist applicationdisplays the content of the event detail display area RTVC2 and thecontent of one of the selected vehicle detail display areas RTVC3 andRTVC3 a of FIG. 8 or FIG. 9 instead of the content of the person detaildisplay area PSDTL1 a. As a result, the operator can confirm thesnapshot of the scene captured by the camera showing the vehicle of thethumbnail THM11, the vehicle license plate of the thumbnail THM11, anddetails such as vehicle type and vehicle color at a glance.

FIG. 11 is a diagram showing an example of a search result screen of theAND search displayed on the client terminal VW1. The search resultscreen WD7 shown in FIG. 11 shows a search result of an AND search forlogically integrating the results of individually searching a pluralityof different objects (specifically, a person and a vehicle).Specifically, the search result screen WD7 includes at least the inputfield CON3 for various search conditions regarding a plurality ofdifferent objects (for example, vehicle) shown in the captured videodata, a display area RTPSVC1 of thumbnails THM21 and THM22 in which atleast one person and one vehicle included in the search result appear,and a person vehicle detail display area PSVSDTL1.

In the input field CON3 for search conditions, both the person searchcondition CH1 (see FIG. 4) and the vehicle search condition CH2 (seeFIG. 5) are selected and set by the operation of the operator in orderto search for each of a plurality of different objects (specifically,persons and vehicles). Further, in the input field CON3 of FIG. 11, atab option TB1 capable of switching between an AND search and an ORsearch is displayed. This tab option TB1 may be displayed in the inputfield CON3 in FIG. 10. For example, in the person search condition CH1in FIG. 11, the object feature elements “male”, “wearing blacklong-sleeved clothes for upper body”, “wearing gray pants for lowerbody”, “no bag”, and “no accessories” are selected. In the vehiclesearch condition CH2 of FIG. 11, the object feature elements of “Sedan”,“Van”, “SUV (Sports Utility Vehicle)”, and “orange” are selected.

Upon detecting that one of the thumbnails THM21 and THM22 (for example,the thumbnail THM21) showing the person and the vehicle has beenselected (see the thick arrow in FIG. 11), as in the case of FIG. 6, theinvestigation assist application requests the video management server 40for captured video data satisfying the identification information of thecamera corresponding to the thumbnail THM 21 and the captured date andtime. Upon receiving the captured video data (that is, the data of theselected vehicle video SMV2) sent from the video management server 40 inresponse to this request, the investigation assist application displaysthe selected person vehicle video SMV2 in the person vehicle detaildisplay area PSVSDTL1. Further, the investigation assist applicationdisplays the installation location CMPS1 of the camera corresponding tothe thumbnail THM21 and a snapshot CPIMG2 of an image of one scene ofthe captured video data of the camera of the installation location CMPS1on the road map data MP1 in a superimposed manner in the person detaildisplay area PSVSDTL1. Thereby, the operator can easily confirm thesnapshot of the scene captured by the camera showing both the person andthe vehicle of the thumbnail THM21 and the installation location CMPS1of the camera.

Next, an operation procedure example assuming an investigation scenarioof the investigation assist system 1 according to Embodiment 1 will bedescribed with reference to FIGS. 12 and 13, respectively. FIG. 12 is asequence diagram showing an operation procedure example in time seriesregarding a first investigation scenario in the investigation assistsystem according to Embodiment 1. FIG. 13 is a sequence diagram showingan operation procedure example in time series regarding a secondinvestigation scenario in the investigation assist system according toEmbodiment 1.

In the first investigation scenario, an example in which powerfuleyewitness information is obtained for each of a person such as asuspect and a vehicle such as a getaway vehicle, and the person orvehicle shown in the captured video data of the camera is individuallysearched by using the eyewitness information as a search condition, andthen the processing result of the search is displayed on the clientterminal VW1 or the mobile terminal VW2 will be described. Hereinafter,for simplification of the description, an example in which a searchprocessing request is sent from the client terminal VW1 to theAI-integration server 10 will be described, but a search processingrequest may be sent from the mobile terminal VW2 as well.

In FIG. 12, the client terminal VW1 generates a processing request foran OR search (see FIG. 10) including the person search condition CH1characterizing a person and the vehicle search condition CH2characterizing a vehicle by the operation of the operator and send therequest to the AI-integration server 10 (SU). Here, the person searchcondition CH1 is, for example, a person whose upper body is wearing blueclothes. The vehicle search condition CH2 is, for example, a vehiclewhose type is sports utility vehicle (SUV).

Upon receiving the processing request of an AND search from the clientterminal VW1, the AI-integration server 10 first takes out the personsearch condition CH1 from the processing request sent in step St1 andsends a processing request for searching for a person who satisfies theperson search condition CH1 to the person search server 60 (St2).

Based on the processing request from the AI-integration server 10, theperson search server 60 refers to the database 62 and executes thesearch processing of a person that satisfies the person search conditionCH1 (St3). When the person search server 60 extracts a person whomatches the characteristics of the person search condition CH1 (St3,match), the person search server 60 returns a processing result (seeabove) including the thumbnail of the person to the AI-integrationserver 10 (St4). On the other hand, when the person search server 60cannot extract a person who matches the characteristics of the personsearch condition CH1 (St3, mismatch), the person search server 60returns a processing result indicating that there is no correspondingperson information to the AI-integration server 10 (St5).

Following the step St4 or step St5, the AI-integration server 10 takesout the vehicle search condition CH2 from the processing request sent instep St1 and sends a processing request for searching for a vehiclesatisfying the vehicle search condition CH2 to the vehicle search server80 (St6).

Based on the processing request from the AI-integration server 10, thevehicle search server 80 refers to the database 82 and executesprocessing of searching for a vehicle satisfying the vehicle searchcondition CH2 (St7). When the vehicle search server 80 extracts avehicle matching the characteristics of the vehicle searching conditionCH2 (St7, match), the vehicle search server 80 returns a processingresult including the thumbnail of the vehicle (see above) to theAI-integration server 10 (St8). On the other hand, when the vehiclesearch server 80 cannot extract the vehicle that matches thecharacteristics of the vehicle search condition CH2 (St7, mismatch), thevehicle search server 80 returns a processing result indicating thatthere is no corresponding vehicle information to the AI-integrationserver 10 (St9).

The AI-integration server 10 integrates (consolidates) the result of theperson search by the person search server 60 (person information) andthe result of the vehicle search by the vehicle search server 80(vehicle information), including the result that there is nocorresponding person and the result that there is no correspondingvehicle, and returns the result to the client terminal VW1 (St10). Theclient terminal VW1 generates a search result screen showing theprocessing result of the search returned in step St10 and displays thescreen on the display device 116 (see FIGS. 6, 8, 10, and 11).

Although FIG. 12 shows the example of an OR search of the person searchand the vehicle search using the person search condition CH1 and thevehicle search condition CH2, the AI-integration server 10 performs thefollowing when performing an AND search of the person search and thevehicle search. For example, the AI-integration server 10 sets theprocessing results of the person search sent in step St4 as thepopulation of the subsequent vehicle search, and causes the vehiclesearch server 80 to search for a processing result that satisfies thevehicle search condition CH2 among the processing results of the personsearch.

In the second investigation scenario, like the first investigationscenario, an example in which each person or vehicle shown in thecaptured video data of the camera is individually searched, and the faceimage of a person is specifically specified (narrowed down) from theprocessing result of the search using the face authentication server 50,and the processing result is displayed on the client terminal VW1 or themobile terminal VW2 will be described.

In FIG. 13, the client terminal VW1 acquires a processing resultincluding thumbnails of a person, a vehicle, or both as processingsubsequent to FIG. 12, and displays the processing result on the displaydevice 116. The client terminal VW1 receives a selection of the clearestthumbnail selected by the operation of the operator from the thumbnailsof the person and the vehicle displayed on the display device 116(St11). The selection in step St11 may be performed, for example, by theoperation of the operator, when it is detected by the function of theinvestigation assist application that a frame is drawn on the face inthe selected thumbnail, or all thumbnails displayed on the displaydevice 116 may be selected at random. Here, in order to simplify thedescription, it is assumed that a thumbnail in which a frame is drawn isselected in step St11 by the operation of the operator.

The client terminal VW1 generates a registrant verification instructionwith the thumbnail selected in step St11 (for example, the face imagecut out by the investigation assist application by selecting the facepart of the person in question by the operator) attached, and sends theinstruction to the AI-integration server 10 (St12). Upon receiving theregistrant verification instruction from the client terminal VW1, theAI-integration server 10 sends the registrant verification instructionwith the thumbnail attached to the face authentication server 50 (St13).

The face authentication server 50 refers to the blacklist data in thedatabase 52 based on the registrant verification instruction from theAI-integration server 10, and searches for a registrant that matches thethumbnail (for example, face image) included in the registrantverification instruction (St14). When the face authentication server 50extracts a face image that matches the thumbnail face image (St14,match), the face authentication server 50 returns a processing resultincluding the target person information (for example, the face image andpersonal information) to the AI-integration server 10 (St15). TheAI-integration server 10 sends the processing result returned from theface authentication server 50 to the client terminal VW1 (St16). Theclient terminal VW1 generates a search result screen showing theprocessing result of the search sent in step St16 and displays thescreen on the display device 116 (see FIGS. 7 and 9).

On the other hand, when the face authentication server 50 cannot extractthe face image that matches the face image of the thumbnail (St14,mismatch), the face authentication server 50 returns a processing resultindicating that there is no corresponding person information to theAI-integration server 10 (St17).

Following step St15 or step St17, the AI-integration server 10 sends tothe face authentication server 50 a processing request for a search fora face image that matches the face image of the same thumbnail usingdata other than blacklist data (St18).

The face authentication server 50 refers to the analysis result of thecaptured video data recorded in the past other than the blacklist dataof the database 52 based on the processing request from theAI-integration server 10, and searches for a registrant that matches athumbnail (for example, face image) included in the registrantverification instruction (St19). When the face authentication server 50extracts a face image that matches the thumbnail face image (St19,match), the face authentication server 50 returns a processing resultincluding the target person information (for example, the face image andpersonal information) to the AI-integration server 10 (St20). TheAI-integration server 10 sends the processing result returned from theface authentication server 50 to the client terminal VW1 (St21). Theclient terminal VW1 generates a search result screen showing theprocessing result of the search sent in step St21 and displays thescreen on the display device 116 (see FIGS. 7 and 9).

On the other hand, when the face authentication server 50 cannot extractthe face image that matches the face image of the thumbnail (St19,mismatch), the face authentication server 50 returns a processing resultindicating that there is no corresponding person information to theAI-integration server 10 (St22). The AI-integration server 10 may returnthe processing result indicating that there is no corresponding personinformation to the client terminal VW1 (St23). The execution of theprocessing of step St23 may be omitted.

As described above, the investigation assist system 1 according toEmbodiment 1 includes a plurality of servers (for example, the faceauthentication server 50, the person search server 60, the vehiclesearch server 80, and the terminals (for example, the client terminalVW1 and the mobile terminal VW2)) and the AI-integration server 10communicatively connected to the plurality of servers. Upon receivingthe captured video data of each of the plurality of cameras C1 to C20,each of the servers performs a video analysis of an object (for example,a person or a vehicle) different from other servers with respect to anincident or the like. The AI-integration server 10 sends a searchrequest for a corresponding object to the servers corresponding to theobject feature element based on the input of a plurality of differentobject feature elements (for example, blue clothes for upper body,vehicle is SUV) from the terminal, and receives the search results ofthe corresponding object from the respective servers to integrate anddisplay the search results on the terminal.

As a result, the investigation assist system 1 can perform across-sectional search using the characteristics of each of a pluralityof different objects shown in the captured video data as searchconditions. Therefore, the investigation assist system 1 can quickly andefficiently assist the specification of a suspect who has caused anincident or the like, and a getaway vehicle used by the suspect forescape, and improve the convenience of an investigation by aninvestigation agency such as the police.

In addition, the server performs a video analysis of the correspondingobject by using an object search algorithm (for example, a person searchalgorithm, a vehicle search algorithm) different from other servers. TheAI-integration server 10 has a common interface (for example, a commonsearch algorithm) in which the object search algorithm used in each ofthe plurality of servers is generalized, and uses this common searchalgorithm to send a search request for the corresponding object to theserver corresponding to the object feature element. As a result, even ifthe makers of the respective servers (specifically, the faceauthentication server 50, the person search server 60, and the vehiclesearch server 80) are different, the investigation assist system 1 canperform a cross-sectional search in which a plurality of objects (forexample, a person and a vehicle) are mixed in response to a singlesearch processing request from the client terminal VW1 or the mobileterminal VW2, and improve the convenience of the operator or the like.

Further, the search result of the corresponding object displayed on theterminal is a thumbnail of each of the plurality of persons. As aresult, the operator or the like can easily and visually grasp thethumbnail showing the entire image of the person who is a candidate ofthe suspect such as the incident on the client terminal VW1 or themobile terminal VW2.

In addition, based on selection of one of the thumbnails, theAI-integration server 10 causes the terminal to display a viewing screenof a captured video (the selected person video SMV1) of the person shownin the selected thumbnail and a map display screen (the road map dataMP1) on which the installation location of the camera corresponding tothe captured video is superimposed (see FIGS. 6 and 7). Thereby, theoperator or the like can easily understand at a glance where in theselected person video SMV1 the person of the thumbnail THM1 is present,and further, the installation location of the camera CMPS1 where theperson of the thumbnail THM1 is shown.

In addition, the plurality of servers include the face authenticationserver 50 that uses a face database (for example, blacklist data) inwhich the faces of persons are registered to perform verification. TheAI-integration server 10 sends to the face authentication server 50 averification request for the face of the person shown in the thumbnailbased on selection of at least one of the thumbnails of the plurality ofpersons, and sends a verification result from the face authenticationserver 50 to the terminal. As a result, the operator or the like caneasily grasp the face image and the personal information of the personin question in the thumbnails of a plurality of persons who can becandidates as the suspect in the incident.

Further, the search result of the corresponding object displayed on theterminal is a thumbnail of each of the plurality of vehicles. As aresult, the operator or the like can easily and visually grasp, on theclient terminal VW1 or the mobile terminal VW2, a thumbnail showing anoverall image of a vehicle that is a candidate for a getaway vehicle onwhich a suspect in an incident or the like is riding.

Further, based on selection of one of the thumbnails, the AI-integrationserver 10 causes the terminal to displays a viewing screen of thecaptured video of the vehicle shown in the selected thumbnail (seeselected vehicle video LPcap1) and a vehicle screen showing detailedvehicle information including the license plate of the vehicle (theimage LPcap2 of the license plate) (see FIGS. 8 and 9). As a result, theoperator or the like can finely confirm the features of the getawayvehicle in the selected vehicle video LPcap1 and easily grasp theinformation of the license plate at a glance.

Further, the plurality of servers include a license authenticationserver (for example, LPR server 90) that uses a vehicle owner database(for example, license plate list data) in which the face image andpersonal information of the vehicle owner are registered in associationwith the license plate, for verification. The AI-integration server 10sends to the LPR server 90 a verification request for the ownerinformation of the vehicle shown in the thumbnail based on selection ofat least one of the thumbnails of the plurality of vehicles, and sends averification result from the LPR server 90 to the terminal. As a result,the operator or the like can confirm the face image and the personalinformation of the owner of the selected vehicle video LPcap1 in detailon the search result screen (see FIG. 9), and can quickly confirm thewhereabouts of the owner.

Further, the AI-integration server 10 sends a search request for thecorresponding object to the server corresponding to the object featureelement based on the input of a plurality of different object featureelements from the terminal, receives the search result of thecorresponding object from each server, and displays the search result ofone corresponding object on the terminal. As a result, the investigationassist system 1 can efficiently narrow down thumbnails showing both aperson and a vehicle in the captured video data of the camera, and caneffectively assist the operator or the like in grasping on the clientterminal VW1 or the mobile terminal VW2 simply and intuitively. Forexample, when both a person and a vehicle are shown in one frame thatconstitutes a captured video, in order to search for such a frame withan AI engine or the like, it is necessary to prepare a large number offrames (captured images) that serve as correct training data, andmachine learning is also complicated. However, according to theinvestigation assist system 1 according to Embodiment 1, since it ispossible to perform a logical AND search of both searches afterindividually performing a person search and a vehicle search, it isexpected that the need for the complicated machine learning describedabove will be eliminated and the efficiency of system building will beincreased.

Embodiment 2

In Embodiment 2, triggered by the behavior detection server 70 detectinga predetermined action caused by at least one person, the AI-integrationserver 10 receives a search processing request from the client terminalVW1 or the mobile terminal VW2. Since the configuration of theinvestigation assist system 1 according to Embodiment 2 is the same asthe configuration of the investigation assist system 1 according toEmbodiment 1, the same reference numerals are given to the samecomponents to simplify or omit the description, and different contentswill be described. In Embodiment 2, the behavior detection server 70 isillustrated as a server that generates an alarm notification and sendsthe alarm notification to the AI-integration server 10, but as describedin Embodiment 1 described above, the following description may beapplied to the alarm notification generated when the correspondingobject (for example, a face, a person, a vehicle, and a license plate)is detected by the server (for example, the face authentication server50, the person search server 60, the vehicle search server 80, and theLPR server 90) other than the behavior detection server 70 during thevideo analysis.

As a result of the video analysis of the captured video data of each ofthe cameras C1 to C20, the behavior detection server 70 generates analarm notification when the predetermined action (see above) isdetected, and sends the alarm notification to the AI-integration server10. The timing when the alarm notification is sent to the AI-integrationserver 10 is a normal monitoring time when the captured video data ofeach of the cameras C1 to C20 is sent to various servers (specifically,the face authentication server 50, the person search server 60, thebehavior detection server 70, and the vehicle search server 80), and thealarm notification is suddenly sent to the AI-integration server 10during the monitoring. Here, the alarm notification includes the content(type) of the predetermined action, and the captured date and time andthe identification information of the camera corresponding to thecaptured video data in which the predetermined action is detected. TheAI-integration server 10 sends the alarm notification suddenly sent fromthe behavior detection server 70 to the client terminal VW1 or themobile terminal VW2.

FIG. 14 is a diagram showing an example of an alarm monitoring screendisplayed on the client terminal VW1. Similar to Embodiment 1, an alarmmonitoring screen WD8 may be displayed on the mobile terminal VW2. Inthe alarm monitoring screen WD8 shown in FIG. 14, for example, thecaptured video data of each of the plurality of cameras is received anddisplayed on the client terminal VW1 via the AI-integration server 10 orthe video management server 40. The alarm monitoring screen WD8 maydisplay not only the display screen of the captured video data of thecameras but also map data MP2 (for example, map data on campus premises)with which the locations where the respective cameras are installed canbe identified. On the map data MP2 of FIG. 14, for example, iconsindicating the installation locations of four cameras CMPS11, CMPS12,CMPS13, and CMPS14 are displayed.

Upon detecting a fight between two persons as a predetermined action,the behavior detection server 70 generates an alarm notificationincluding information indicating that there was a fight, the captureddate and time corresponding to a captured video data ALM1 in which thefight was detected, and the identification information of the cameraCMPS11, and sends the alarm notification to the AI-integration server10. In the alarm monitoring screen WD8 of FIG. 14, the captured videodata ALM1 in which a fight has been detected is highlighted by theclient terminal VW1, the icon of the camera CMPS11 is highlighted (forexample, displayed in red and bold), further, a snapshot CPIMG11(captured image) showing one scene of the captured video of the cameraCMPS11 is also displayed in a superimposed manner. Further, in the alarmmonitoring screen WD8 of FIG. 14, every time a predetermined action isdetected by the behavior detection server 70, the client terminal VW1displays an event list EVLST1 regarding the occurrence of an action (anexample of an event) in a superimposed manner. The event list EVLST1shows the date and time when the action has been detected by thebehavior detection server 70 (Date), the identification information ofthe camera that has captured the captured video data used to detect theaction, and the content of an event (action) (for example, a fight) inassociation with each other.

Next, an example of an operation procedure of an image search using thelive video or the past recorded video of the investigation assist system1 according to Embodiment 2 will be described with reference to FIGS. 15to 17, respectively. FIG. 15 is a sequence diagram showing an operationprocedure example in time series regarding an image search using livevideo in the investigation assist system 1 according to Embodiment 2.FIGS. 16 and 17 are sequence diagrams showing an operation procedureexample in time series regarding an image search using the past recordedvideo in the investigation assist system 1 according to Embodiment 2. Inthe description of FIGS. 16 and 17, the same content as the processingdescribed in FIG. 15 is given the same step number to simplify or omitthe description, and different content will be described.

In FIG. 15, upon detecting a fight between two persons as apredetermined action, the behavior detection server 70 generates analarm notification including information indicating that there was afight, the captured date and time corresponding to a captured video dataALM1 in which the fight was detected, and the identification informationof the camera CMPS11, and sends the alarm notification to theAI-integration server 10 (St31). The AI-integration server 10 sends thealarm notification suddenly sent from the behavior detection server 70to the client terminal VW1 (St32).

Based on the reception of the alarm notification sent from theAI-integration server 10, the investigation assist application of theclient terminal VW1 highlights the captured video data ALM1 (forexample, the red frame shown in FIG. 14) which is a live video of thecamera corresponding to the identification information of the cameraincluded in the alarm notification (St33). The policeman who is the userof the mobile terminal VW2 rushes to the scene where two persons getinto a fight based on the receipt of the alarm notification (St34). Theclient terminal VW1 extracts a captured image (snapshot) showing aperson (for example, two persons get into a fight) who is the target ofan alarm notification from the captured video data ALM1 which is thetarget of an alarm notification (St35). Further, the client terminal VW1cuts out the faces of the two persons get into a fight from the capturedimage (snapshot) extracted in step St35 (St36). The following processingis executed for each of the two persons, but for simplicity ofdescription, processing for one of the two face images will bedescribed.

The client terminal VW1 generates a registrant verification instructionwith the face image of the person cut out in step St36 attached andsends the instruction to the AI-integration server 10 (St37). Uponreceiving the registrant verification instruction from the clientterminal VW1, the AI-integration server 10 sends the registrantverification instruction with the thumbnail attached to the faceauthentication server 50 (St38).

The face authentication server 50 refers to the blacklist data in thedatabase 52 based on the registrant verification instruction from theAI-integration server 10, and searches for a registrant that matches thethumbnail (for example, face image) included in the registrantverification instruction (St39). When the face authentication server 50extracts a face image that matches the cut-out face image (St39, match),the face authentication server 50 returns a processing result includingthe target person information (for example, the face image and personalinformation) to the AI-integration server 10 (St40). The AI-integrationserver 10 sends the processing result returned from the faceauthentication server 50 to the client terminal VW1 (St41). The clientterminal VW1 generates a search result screen showing the processingresult of the search sent in step St41 and displays the screen on thedisplay device 116 (see FIG. 14).

On the other hand, when the face authentication server 50 cannot extractthe face image that matches the face image of the thumbnail (St39,mismatch), the face authentication server 50 returns a processing resultindicating that there is no corresponding person information to theAI-integration server 10 (St42).

Following step St41 or step St42, the AI-integration server 10 sends tothe face authentication server 50 a processing request for a search fora face image that matches the face image of the same thumbnail usingdata other than blacklist data (St43).

The face authentication server 50 refers to the analysis result of thecaptured video data recorded in the past other than the blacklist dataof the database 52 based on the processing request from theAI-integration server 10, and searches for a registrant who matches theface image included in the registrant verification instruction (St44).When the face authentication server 50 extracts a face image thatmatches the face image (St44, match), the face authentication server 50returns a processing result including the target person information (forexample, a face image and personal information) to the AI-integrationserver 10 (St45). The AI-integration server 10 sends the processingresult returned from the face authentication server 50 to the clientterminal VW1 (St46). The client terminal VW1 generates a search resultscreen showing the processing result of the search sent in step St45 anddisplays the screen on the display device 116 (see FIG. 14).

On the other hand, when the face authentication server 50 cannot extractthe face image that matches the face image (St44, mismatch), the faceauthentication server 50 returns a processing result indicating thatthere is no corresponding person information to the AI-integrationserver 10 (St47). The AI-integration server 10 may return the processingresult indicating that there is no corresponding person information tothe client terminal VW1 (St48). The execution of the processing of stepSt48 may be omitted.

In FIG. 16, based on the reception of the alarm notification sent fromthe AI-integration server 10, the investigation assist application ofthe client terminal VW1 searches for the same alarm notification as thealarm notification sent from the AI-integration server 10 in step St32among a plurality of previously received alarm notifications (alarmevents) recorded in the recording device 112 of the client terminal VW1(St51). The client terminal VW1 reads, from the recording device 112,the captured video data which is the past recorded video of the cameracorresponding to the identification information of the camera includedin the past alarm notification obtained by the search in step St51, andreproduces and displays the data (St52). The client terminal VW1extracts a captured image (snapshot) showing a person (for example, twopersons get into a fight) who is the target of the alarm notificationfrom the past captured video data reproduced in step St52 (St35 a).Further, when the face (for example, a face facing the front, suitablefor face authentication) of the person who is taking the predeterminedaction (that is, the action included in the past alarm notificationobtained by the search in step St51) is shown in the captured image(snapshot) extracted in step St35 a, the client terminal VW1 cuts outthe face image (St36). The processing in steps St37 to St48 is the sameas that in FIG. 16, and thus description thereof will be omitted.

In FIG. 17, when the investigation assist application of the clientterminal VW1 cannot extract the face of a person (for example, a facefacing the front, suitable for face authentication) who is taking thepredetermined action (that is, the action included in the past alarmnotification obtained by the search in step St51) from the capturedimage (snapshot) extracted in step St35 a (St36 a), the investigationassist application cuts out a full-body image of the person from thesnapshot (St53).

The client terminal VW1 generates a person re-verification instructionwith the person image cut out in step St53 attached, and sends theinstruction to the AI-integration server 10 (St54). Upon receiving theperson re-verification instruction from the client terminal VW1, theAI-integration server 10 sends the person re-verification instructionwith the person image attached to the person search server 60 (St55).

The person search server 60 refers to the video analysis result of thecaptured video data stored in the database 62 based on the personre-verification instruction from the AI-integration server 10, andsearches for a person who matches or is similar to the person imageincluded in the person re-verification instruction (St56). When a personimage that matches or is similar to the person image is extracted (St56,similar), the person search server 60 returns a processing resultincluding the target person information (for example, the thumbnail ofthe target person) to the AI-integration server 10 (St57). TheAI-integration server 10 sends the processing result returned from theperson search server 60 to the client terminal VW1 (St58). The clientterminal VW1 generates a search result screen showing the processingresult of the search sent in step St58 and displays the screen on thedisplay device 116.

On the other hand, when the person search server 60 cannot extract anyface image that matches or is similar to the person image (St56, notsimilar), the person search server 60 returns a processing resultindicating that there is no corresponding person information to theAI-integration server 10 (St59). The AI-integration server 10 displaysthe processing result of step St59 on the client terminal VW1.

As described above, in the investigation assist system 1 according toEmbodiment 2, the plurality of servers include the behavior detectionserver 70 that detects a predetermined action caused by at least oneperson based on the videos captured by the plurality of cameras. Basedon the action detection, the behavior detection server 70 sends an alarmnotification including camera information (for example, identificationinformation of a camera) of the camera corresponding to the capturedvideo data in which the action is detected to the AI-integration server10. As a result, when a predetermined action is detected whilemonitoring the location where each of the plurality of cameras isinstalled, since the investigation assist system 1 can receive the alarmnotification from the behavior detection server 70, it is possible topromptly grasp the location where the action has been taken and toappropriately monitor the captured video data of the camera at thatlocation.

In addition, the plurality of servers include the face authenticationserver 50 that uses a face database (for example, blacklist data) inwhich the faces of persons are registered to perform verification. TheAI-integration server 10 sends to the terminal an instruction to displaya live video corresponding to the camera information included in thealarm notification. The AI-integration server 10 sends a verificationrequest for the face of a person to the face authentication server 50based on the selection of the person shown in the live video displayedon the terminal based on the display instruction, and sends averification result from the face authentication server 50 to theterminal. As a result, since the investigation assist system 1 canspecify a person from the face image of the person shown in the livevideo of the camera capturing the scene when the occurrence of an actionis detected, it is possible to efficiently detect a person who can be asuspect in an incident at an early stage.

Further, the face authentication server 50 verifies the face of a personby using a face database (for example, blacklist data) based on theverification request for the face of the person, and sends averification result to the AI-integration server 10. As a result, theinvestigation assist system 1 can specify the face image and thepersonal information of the person who has taken the action at an earlystage, and can improve the work efficiency of the police officer and thelike.

Further, the face authentication server 50 further verifies the face ofa person by using the captured video data of each of the plurality ofcameras based on the verification request for the face of the person,and sends a verification result to the AI-integration server 10. As aresult, the investigation assist system 1 can specify with high accuracya person shown in the captured video data of the camera capturing thescene when an action is detected.

Further, the predetermined action is at least one of staggering, afight, possession of a pistol, and shoplifting. As a result, theinvestigation assist system 1 can appropriately detect an actionequivalent to at least one crime among staggering, a fight, possessionof a pistol, and shoplifting while monitoring a location where each of aplurality of cameras is installed, and it is possible to assist theearly specification of suspects such as incidents by police officers.

Although various embodiments have been described with reference to thedrawings, it goes without saying that the present disclosure is notlimited to such examples. It is obvious to those skilled in the art thatvarious changes, modifications, substitutions, additions, deletions andequivalents can be conceived within the scope of the claims, and it isunderstood that of course those also belong to the technical scope ofthe present disclosure. Further, the respective constituent elements inthe various embodiments described above may be arbitrarily combinedwithout departing from the spirit of the invention.

In Embodiment 1 described above, as the search handled by theAI-integration server 10, a person search by the person search server 60and a vehicle search by the vehicle search server 80 have been describedas examples, but the search is not limited thereof. In other words, asshown in FIG. 4 or 5, the search condition is not limited to a person ora vehicle. For example, the search handled by the AI-integration server10 may be applied to face detection (face search) by the faceauthentication server 50, a license plate search by the LPR server 90,and further, a person search by the person search server 60 and avehicle search by the vehicle search server 80 may be used incombination. In this case, a face image (image search) or a licenseplate (text search or image search) may be used as the search condition.As a search result using a face image, for example, the face image FCE1shown in FIG. 7 may be displayed. As a search result using a licenseplate, for example, the license plate detail display information PLT1shown in FIG. 8 or 9 and the face image FCE2 of the purchaser of thevehicle corresponding to the license plate information may be displayed.

In the above-described Embodiment 1, as an example of the AND search(cross-sectional search) handled by the AI-integration server 10, theAND search of a person search by the person search server 60 and avehicle search by the vehicle search server 80, and the AND search of aface verification by the face authentication server 50, a person searchby the person search server 60, and a vehicle search by the vehiclesearch server 80 have been described, but examples of the AND search arenot limited thereto. For example, the AI-integration server 10 canperform an AND search (cross-sectional search) with the followingcombinations.

For example, in addition to face verification by the face authenticationserver 50, at least one of the person search by the person search server60, the behavior detection by the behavior detection server 70 (forexample, verification of face images similar to those of persons who getinto a fight or have a pistol), the vehicle search by the vehicle searchserver 80 (for example, verification of face images of males in theirthirties in a white car), the license plate verification by the LPRserver 90 (for example, verification of face images of persons who arein a vehicle with a specific number in the upper two digits and aresimilar to a certain face image), and the like can be combined.

Further, for example, in addition to the person search by the personsearch server 60, at least one of the behavior detection by the behaviordetection server 70 (for example, searching for a person in blackclothes carrying vandalism or possessing a pistol), the search by thevehicle search server 80 (for example, searching for a person in redclothes approaching a white car), the license plate verification by theLPR server 90 (for example, searching for a person in black clothesapproaching a vehicle with a specific number), and the like can becombined.

Further, for example, in addition to the vehicle search by the vehiclesearch server 80, at least one of the behavior detection by the behaviordetection server 70 (for example, searching for a sedan-type vehiclerunning in a reverse direction or a taxi vehicle threatened by a pistolfrom the outside) and the license plate verification by the LPR server90 (for example, searching for a vehicle running in a reverse directionwith a specific number in the upper two digits) can be combined.

Further, for example, in addition to license plate verification by theLPR server 90, at least one of the behavior detection by the behaviordetection server 70 (for example, detection of an action of running in areverse direction with a specific number in the upper two digits,detection of an action of threatening the driver of a taxi vehicle witha specific number in the upper two digits with a pistol from theoutside) and the like can be combined.

Embodiment 1 described above illustrates that the object in each of theimages captured by the cameras C1 to C20 is a person or a vehicle, butthe object is not limited to a person or a vehicle, and may be anotherobject (for example, a moving body). The moving body may be a flyingobject such as a drone operated by a person such as a suspect who hascaused an incident.

The present disclosure is useful as an investigation assist system, aninvestigation assist method, and a computer program that improve theconvenience of an investigation by an investigation agency such as thepolice by promptly and efficiently assisting the specification of asuspect who has caused an incident or a getaway vehicle used by thesuspect for escape.

The present application is based upon Japanese Patent Application(Patent Application No. 2019-160659 filed on Sep. 3, 2019), the contentof which is incorporated herein by reference.

What is claimed is:
 1. An investigation assist system comprising: aplurality of servers; and an integration server communicativelyconnected to a terminal and the plurality of servers, wherein inresponse to reception of a video captured by a plurality of cameras,each of the plurality of servers performs a video analysis of an objectwith respect to an incident, the plurality of servers processingdifferent objects, respectively, and based on an input of a plurality ofdifferent object feature elements from the terminal, the integrationserver sends a search request for corresponding objects to therespective servers corresponding to the object feature elements,receives and integrates search results of the corresponding objects fromthe respective servers, and causes the terminal to display an integratedsearch result.
 2. The investigation assist system according to claim 1,wherein each of the plurality of servers performs the video analysis ofthe object based on an object search algorithm, the plurality of serversusing different search algorithms, respectively; and the integrationserver has a common search algorithm in which the object searchalgorithms used in the plurality of servers are generalized, and usesthe common search algorithm to send the search request for thecorresponding objects to the respective servers corresponding to theobject feature elements.
 3. The investigation assist system according toclaim 1, wherein the integrated search result of the correspondingobjects displayed on the terminal comprises thumbnails of a plurality ofpersons, respectively.
 4. The investigation assist system according toclaim 3, wherein based on selection of one of the thumbnails, theintegration server causes the terminal to display a viewing screen of acaptured video of a person shown in the selected one of the thumbnailsand a map display screen on which an installation location of a cameracorresponding to the captured video is superimposed.
 5. Theinvestigation assist system according to claim 3, wherein the pluralityof servers includes a face authentication server that uses a facedatabase in which faces of persons are registered for verification; andbased on selection of at least one of the thumbnails of the plurality ofpersons, the integration server sends to the face authentication servera verification request for a face of a person shown in the selected atleast one of the thumbnails, and sends a verification result from theface authentication server to the terminal.
 6. The investigation assistsystem according to claim 1, wherein the integrated search result of thecorresponding objects displayed on the terminal comprises thumbnails ofa plurality of vehicles.
 7. The investigation assist system according toclaim 6, wherein based on selection of one of the thumbnails, theintegration server causes the terminal to display a viewing screen of acaptured video of a vehicle shown in the selected one of the thumbnails,and a vehicle screen showing detailed vehicle information including alicense plate of the vehicle.
 8. The investigation assist systemaccording to claim 6, wherein the plurality of servers include a licenseauthentication server that uses a vehicle owner database in which faceimages and personal information of vehicle owners are registered inassociation with license plates for verification; and based on selectionof at least one of the thumbnails of the plurality of vehicles, theintegration server sends to the license authentication server averification request for owner information of a vehicle shown in theselected at least one of the thumbnails, and sends a verification resultfrom the license authentication server to the terminal.
 9. Theinvestigation assist system according to claim 1, wherein the pluralityof servers include an behavior detection server that detects apredetermined action caused by at least one person, based on videoscaptured by the plurality of cameras; and the behavior detection serversends to the integration server an alarm notification including camerainformation corresponding to a captured video in which the action isdetected, based on detection of the action.
 10. The investigation assistsystem according to claim 9, wherein the plurality of servers include aface authentication server that uses a face database in which faces ofpersons are registered for verification; and the integration serversends a display instruction for a live video corresponding to the camerainformation included in the alarm notification to the terminal, andbased on selection of a person shown in the live video displayed on theterminal based on the display instruction, sends a verification requestfor a face of the person to the face authentication server, and send averification result from the face authentication server to the terminal.11. The investigation assist system according to claim 10, wherein theface authentication server verifies the face of the person by using theface database based on the verification request for the face of theperson, and sends a verification result to the integration server. 12.The investigation assist system according to claim 11, wherein the faceauthentication server further verifies the face of the person by usingthe captured video of each of the plurality of cameras based on theverification request for the face of the person, and sends averification result to the integration server.
 13. The investigationassist system according to claim 9, wherein the predetermined actioncomprises at least one of staggering, a fight, possession of a pistol,and shoplifting.
 14. The investigation assist system according to claim1, wherein based on an input of the plurality of different objectfeature elements from the terminal, the integration server sends asearch request for corresponding objects to the respective serverscorresponding to the object feature elements, receives search results ofthe corresponding objects from the respective servers, and causes theterminal to display a search result of one of the corresponding objects.15. An investigation assist method performed by an investigation assistsystem including a plurality of servers, and an integration servercommunicatively connected to a terminal and the plurality of servers,the investigating assist method comprising: receiving a video capturedby a plurality of cameras, causing a plurality of servers to perform avideo analysis of an object with respect to an incident, the pluralityof servers processing different objects, respectively, based on an inputof a plurality of different object feature elements from the terminal,sending a search request for corresponding objects to the respectiveservers corresponding to the object feature elements; and receiving andintegrating search results of the corresponding objects from therespective servers, and causing the terminal to display an integratedsearch result.